Skip to content

perf: Optimize NULL handling in arrays_zip#21475

Merged
mbutrovich merged 2 commits intoapache:mainfrom
neilconway:neilc/perf-arrays-zip-nulls
Apr 10, 2026
Merged

perf: Optimize NULL handling in arrays_zip#21475
mbutrovich merged 2 commits intoapache:mainfrom
neilconway:neilc/perf-arrays-zip-nulls

Conversation

@neilconway
Copy link
Copy Markdown
Contributor

@neilconway neilconway commented Apr 8, 2026

Which issue does this PR close?

Rationale for this change

arrays_zip was building a Vec<bool> and then converting it to a NullBuffer. It is simpler and (a bit) faster to just use NullBuffer / NullBufferBuilder directly.

Benchmarks (ARM64):

  - arrays_zip_no_nulls_8192: 1096.5µs → 1010.7µs, -7.8%
  - arrays_zip_10pct_nulls_8192: 1131.8µs → 1100.1µs, -2.8%

The improvement is not massive but it's non-zero, and the resulting code is cleaner and more idiomatic.

What changes are included in this PR?

  • Implement optimization
  • Add benchmark

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

Copy link
Copy Markdown
Contributor

@mbutrovich mbutrovich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me. Seems more idiomatic with Arrow-rs, and we'll take performance wins! Thanks @neilconway!

@mbutrovich mbutrovich added this pull request to the merge queue Apr 10, 2026
Copy link
Copy Markdown
Contributor

@metegenez metegenez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a 50% null density case to the benchmark? I am not sure we are firing the all-null-row early exit.

LGTM.

Merged via the queue into apache:main with commit fbb5240 Apr 10, 2026
35 checks passed
@neilconway neilconway deleted the neilc/perf-arrays-zip-nulls branch April 10, 2026 20:44
alamb added a commit to apache/arrow-rs that referenced this pull request Apr 14, 2026
…on_many` (#9692)

## Which issue does this PR close?

- Closes #8809.

## Rationale for this change

Several DataFusion PRs
([#21464](apache/datafusion#21464),
[#21468](apache/datafusion#21468),
[#21471](apache/datafusion#21471),
[#21475](apache/datafusion#21475),
[#21477](apache/datafusion#21477),
[#21482](apache/datafusion#21482),
[#21532](apache/datafusion#21532)) optimize NULL
handling in scalar functions by replacing row-by-row null buffer
construction with bulk `NullBuffer::union`. When 3+ null buffers need
combining, they chain binary `union` calls, each allocating a new
`BooleanBuffer`.

`NullBuffer::union_many` reduces this to 1 allocation (clone + in-place
ANDs). For example, from
[#21482](apache/datafusion#21482):

Before:
```rust
[array.nulls(), from_array.nulls(), to_array.nulls(), stride.and_then(|s| s.nulls())]
    .into_iter()
    .fold(None, |acc, nulls| NullBuffer::union(acc.as_ref(), nulls))
```
After:
```rust
NullBuffer::union_many([
    array.nulls(),
    from_array.nulls(),
    to_array.nulls(),
    stride.and_then(|s| s.nulls()),
])
```

Per @alamb's
[suggestion](#9692 (comment)),
this PR also implements the general-purpose mutable bitwise operations
on `BooleanArray` from #8809, following the `PrimitiveArray::unary` /
`unary_mut` pattern. This builds on the
`BitAndAssign`/`BitOrAssign`/`BitXorAssign` operators added to
`BooleanBuffer` in #9567.

## What changes are included in this PR?

**`NullBuffer::union_many(impl IntoIterator<Item =
Option<&NullBuffer>>)`**: combines multiple null buffers in a single
allocation (clone + in-place `&=`). Used by DataFusion for bulk null
handling.

**`BooleanArray` bitwise operations** (6 new public methods):

Unary (`op: FnMut(u64) -> u64`):
- `bitwise_unary(&self, op)` — always allocates a new array
- `bitwise_unary_mut(self, op) -> Result<Self, Self>` — in-place if
uniquely owned, `Err(self)` if shared
- `bitwise_unary_mut_or_clone(self, op)` — in-place if uniquely owned,
allocates if shared

Binary (`op: FnMut(u64, u64) -> u64`):
- `bitwise_bin_op(&self, rhs, op)` — always allocates, unions null
buffers
- `bitwise_bin_op_mut(self, rhs, op) -> Result<Self, Self>` — in-place
if uniquely owned, `Err(self)` if shared, unions null buffers
- `bitwise_bin_op_mut_or_clone(self, rhs, op)` — in-place if uniquely
owned, allocates if shared, unions null buffers

Note: #8809 proposed the binary variants take a raw buffer and
`right_offset_in_bits`. This PR takes `&BooleanArray` instead, which
encapsulates both and matches existing patterns like
`BooleanArray::from_binary`.

## Are these changes tested?

Yes. 23 tests for the `BooleanArray` bitwise methods and 6 tests for
`union_many`, covering:
- Basic correctness (AND, OR, NOT)
- Null handling (both nullable, one nullable, no nulls, null union)
- Buffer ownership (uniquely owned → in-place, shared → `Err` /
fallback)
- Edge cases (empty arrays, sliced arrays with non-zero offset,
misaligned left/right offsets)

## Are there any user-facing changes?

Six new public methods on `BooleanArray` and one new public method on
`NullBuffer`.

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optimize NULL handling in arrays_zip

3 participants